Conditional Expectation

Conditional expectation given a non trivial event

Consider a probability space \((\Omega, \sF, P)\) and an event \(B \in \sF\) such that \(P(B) > 0\). Then, the conditional probability of any event \(A \in \sF\) given an event \(B\) is defined as Consider a random variable \(X: \Omega\to\R\) defined on a probability space \((\Omega, \sF, P)\), with distribution function \(F_X: \R \to [0,1]\), and a non trivial event \(B \in \sF\) such that \(P(B) > 0\).

Definition 1. The conditional distribution of \(X\) given event \(B\) is denoted by \(F_{X| B}: \R \to [0,1]\) and \(F_{X| B}(x)\) is defined as the probability of event \(A_X(x) \triangleq X^{-1}(-\infty, x]\) conditioned on event \(B\) for all \(x\in\R\). That is,

Remark 1. The conditional distribution \(F_{X|B}\) is a distribution function. This follows from the fact that (i) \(F_{X|B} \ge 0\), (ii) \(F_{X|B}\) is right continuous, (iii) \(\lim_{x\downarrow -\infty}F_{X|B}(x) = 0\) and \(\lim_{x\uparrow \infty}F_{X|B}(x) = 1\).

Remark 2. For a discrete random variable \(X:\Omega\to\sX\), the conditional probability mass function of \(X\) given a non trivial event \(B\) is given by \(P_{X|B}(x) = \frac{P(X^{-1}\set{x}\cap B)}{P(B)}\) for all \(x \in \sX\).

Remark 3. For a continuous random variable \(X:\Omega\to\R\), the conditional density of \(X\) given a non trivial event \(B\) is given by \(f_{X|B}(x) = \frac{dF_{X|B}(x)}{dx}\) for all \(x \in \R\).

Example 2 (Conditional distribution). Consider the probability space \((\Omega, \sF, P)\) corresponding to a random experiment where a fair die is rolled once. For this case, the outcome space \(\Omega = [6]\), the event space \(\sF = \cP([6])\), and the probability measure \(P(\omega) = \frac{1}{6}\) for all \(\omega\in \Omega\).

We define a random variable \(X: \Omega\to\R\) such that \(X(\omega) = \omega\) for all \(\omega \in \Omega\), and an event \(B \triangleq \set{\omega \in \Omega: X(\omega) \le 3} = [3] \in \sF\). We note that \(P(B) = 0.5\) and the conditional PMF of \(X\) given \(B\) is

Definition 3. The conditional expectation of \(X\) given event \(B\) is given as \(\E[X|B] \triangleq \int_{x\in\R}xdF_{X|B}(x).\)

Remark 4. For a discrete random variable \(X:\Omega\to\sX\), the conditional expectation of \(X\) given a non trivial event \(B\) is given by \(\E[X|B]= \sum_{x\in\sX}xP_{X|B}(x)\).

Example 4 (Conditional expectation). For the random variable \(X\) and event \(B\) defined in Example [exmp:FairDieConditionHalf], the conditional expectation \(\E[X|B] = 2\).

Remark 5. Consider two random variables \(X,Y\) defined on this probability space, then for \(y \in \R\) such that \(F_Y(y) > 0\), we can define events \(A_X(x) \triangleq X^{-1}(-\infty, x]\) and \(A_Y(y) = Y^{-1}(-\infty, y]\), such that The key observation is that \(\set{Y \le y}\) is a non-trivial event. How do we define conditional expectation based on events such as \(\set{Y = y}\)? When random variable \(Y\) is continuous, this event has zero probability measure.

Conditional expectation given an event space

Consider random variables \(X:\Omega\to\R\) and \(Y:\Omega\to\R\) defined on the same probability space \((\Omega, \sF, P)\) such that \(\E\abs{X} < \infty\), and a smaller event space \(\sG \subset \sF\). For each non trivial event \(G \in \sG\), we know how to define the conditional distribution \(F_{X|G}\) and \(\E[X|G]\). For any trivial event \(N \in \sG\), these are undefined.

Definition 5. The conditional expectation of random variable \(X\) given event space \(\sG\) is a random variable \(\E[X\given \sG]: \Omega \to \R\) defined on the same probability space, such that

\(Z \triangleq \E[X\given \sG]\) is \(\sG\) measurable,

for all \(G\in \sG\), we have \(%\int_AXdP = \E[X\Ind{G}] = \E[Z\Ind{G}], %=\int_AYdP,\)

\(\E\abs{Z}<\infty\).

Lemma 6. The conditional expectation of \(X\) given \(\sG\) is an a.s. unique random variable.

Proof. Proof. Consider two random variables \(Z_1 = \E[X|\sG]\) and \(Z_2 = \E[X|\sG]\). Then from the definition, \(Z_1, Z_2\) are \(\sG\) measurable random variables, and \(Z_1-Z_2\) is also \(\sG\) measurable. Therefore, \(G_n \triangleq \set{Z_1 - Z_2 > \frac{1}{n}} \in \sG\) and \(\E[(Z_1-Z_2)\Ind{G_n}] = 0\) by definition. It follows from continuity of probability, that \(P(\lim_n G_n) = 0\). Similarly, defining \(F_n \triangleq \set{Z_2 - Z_1 > \frac 1 n}\), we can show that \(P(\lim_nF_n) = 0\). ◻

Example 7 (Conditional expectation as averaging). Consider a random variable \(X:\Omega\to\R\) defined on a probability space \((\Omega,\sF,P)\) with \(\E\abs{X} < \infty\), and the coarsest event space \(\sG = \set{\emptyset, \Omega} \subseteq \sF\) and finest event space \(\sF\). We observe that \(\E[X|\sG] = \E X\) a.s. uniquely, since (i) \(\E X\) is a constant and hence \(\sG\) measurable, (ii) \(\E[\E X \Ind{\emptyset}] = \E[X \Ind{\emptyset}] = 0\) and \(\E[\E X \Ind{\Omega}] = \E[X \Ind{\Omega}] = \E X\), and (iii) \(\E\abs{\E X} = \abs{\E X} \le \E \abs{X} < \infty\) from the Jensen’s inequality.

We also observe that \(\E[X|\sF] = X\) a.s. uniquely, since (i) \(X\) is \(\sF\) measurable random variable, (ii) \(\E[ X \Ind{G}] = \E[X \Ind{G}]\) for all events \(G\in \sF\), and (iii) \(\E\abs{X} < \infty\).

Lemma 8. The mean of conditional expectation of random variable \(X\) given event space \(\sG\) is \(\E X\).

Proof. Proof. From the definition of event space \(\Omega\in\sG\), and from the definition of conditional expectation, we get \(\E[\E[X\mid \sG]] = \E[\E[X\mid \sG]\Ind{\Omega}] = \E[X\Ind{\Omega}] = \E X.\) ◻

Definition 9. The conditional expectation of \(X\) given \(Y\) is a random variable \(\E[X\given Y] \triangleq \E[X\given\sigma(Y)]\) defined on the same probability space.

Example 10 (Conditioning on simple random variables). For a simple random variable \(Y: \Omega \to \sY \subseteq \R\) defined on the probability space \((\Omega, \sF, P)\), we define fundamental events \(E_y \triangleq Y^{-1}\set{y} \in \sF\) for all \(y \in \sY\). Then the sequence of events \(E \triangleq (E_y \in \sF: y \in \sY)\) partitions the sample space, and we can write the event space generated by random variable \(Y\) as \(\sigma(Y) = (\cup_{y\in I}E_y: I \subseteq \sY)\).

For a random variable \(X: \Omega \to \R\) defined on the same probability space, the random variable \(Z \triangleq \E[X\given Y]\) is \(\sigma(Y)\) measurable. Therefore, \(\E[X| Y] = \sum_{y\in\sY}\alpha_y\Ind{E_y}\) for some \(\alpha\in\R^\sY\). We verify that \(Z:\Omega\to\R\) is a \(\sigma(Y)\) measurable random variable, since \(\sigma(Z) \subseteq \sigma(Y)\). We can also check that \(\E\abs{Z} < \infty\). Further, we have \(\E[Z\Ind{E_y}] = \E[X\Ind{E_y}]\) for any \(y \in \sY\), which implies that \(\alpha_y = \frac{\E[X\Ind{E_y}]}{P_Y(y)}\) for any \(y \in \sY\). Notice that

Remark 6. There are three main takeaways from this definition. For a random variable \(Y\), the event space generated by \(Y\) is \(\sigma(Y)\).

1. The conditional expectation \(\E[X|Y] = \E[X|\sigma(Y)]\) and is \(\sigma(Y)\) measurable. That is, \(\E[X|Y]\) is a Borel measurable function of \(Y\). In particular when \(Y\) is discrete, this implies that \(\E[X|Y]\) is a simple random variable that takes value \(\E[X|E_y]\) when \(\omega \in E_y\), and the probability of this event is \(P_Y(y)\). When \(Y\) is continuous, \(\E[X|Y]\) is a continuous random variable with density \(f_Y\).

2. Expectation is averaging. Conditional expectation is averaging over event spaces. We can observe that the coarsest averaging is \(\E[X|\set{\emptyset, \Omega}] = \E X\) and the finest averaging is \(\E[X|\sigma(X)] = X\). Further, \(\E[X|\sigma(Y)]\) is averaging of \(X\) over events generated by \(Y\). If we take any event \(A \in \sigma(Y)\) generated by \(Y\), then the conditional expectation of \(X\) given \(Y\) is fine enough to find the averaging of \(X\) when this event occurs. That is, \(\E[X\Ind{A}] = \E[\E[X|Y]\Ind{A}]\).

3. If \(X \in L^1\), then the conditional expectation \(\E[X|Y] \in L^1\).

Conditional distribution given an event space

Definition 11. The conditional probability of an event \(A\in\sF\) given event space \(\sG\) is defined as \(P(A\mid \sG) \triangleq \E[\Ind{A}\mid\sG]\).

Remark 7. From the definition of conditional expectation, it follows that \(P(A\mid \sG):\Omega\to[0,1]\) is a \(\sG\) measurable random variable, such that \(\E[\Ind{G}P(A|\sG)] = P(A\cap G)\) for all \(G \in \sG\), and is uniquely defined up to sets of probability zero.

Example 12. For the trivial sigma algebra \(\sG =\set{\emptyset ,\Omega}\), the conditional probability is the constant function \(P(A\mid \set{\emptyset ,\Omega})=P (A).\)

Example 13. If \(A\in \sG\), then \(P(A\mid\sG)=\Ind{A}\).

Definition 14. The conditional distribution of random variable \(X\) given sub event space \(\sG\) is defined as \(F_{X \mid \sG}(x)\triangleq P(A_X(x)\mid \sG)\text{ for all }x \in \R.\)

Remark 8. Recall that \(F_{X\mid\sG}(x): \Omega\to[0,1]\) a random variable, for each \(x\in\R\). Further, we observe that \(F_{X\mid\sG}\) is monotone nondecreasing in \(x \in \R\), right continuous at all \(x\in\R\), and has limits \(\lim_{x\downarrow-\infty}F_{X\mid\sG}(x) = 0\) and \(\lim_{x\uparrow\infty}F_{X\mid\sG}(x) = 1\). It follows that \(F_{X\mid\sG}: \Omega \to [0,1]^\R\) is a random distribution.

Theorem 15. Let \(g:\R\to\R\) be a Borel measurable function and \(\sG\) be a sub-event space. Then, the conditional expectation \(\E[g(X)\mid\sG] = \int_{x\in\R}g(x)dF_{X\mid\sG}(x).\)

Proof. Proof. It suffices to show this for simple random variables \(X:\Omega\to\sX\). Since \(g\) is Borel measurable, then \(g(X)\) is a random variable. We will show that \(\E[g(X)\mid\sG] = \sum_{x\in\sX}g(x)P_{X\mid\sG}(x)\) by showing that it satisfies three properties of conditional expectation. For part (i), we observe that from the definition of conditional probability \(P_{X\mid\sG}(x)\) is a \(\sG\)-measurable random variable for all \(x\in\sX\), and so is the linear combination \(\sum_{x\in\sX}g(x)P_{X\mid\sG}(x)\). For part (ii), we let \(G \in \sG\). Then, it follows from the linearity of expectation and the definition of conditional probability, that For part (iii), it follows from the triangle inequality, the linearity of expectation, and the definition of conditional probability that \(\E\abs{\sum_{x\in\sX}g(x)P_{X\mid\sG}(x)} \le \sum_{x\in\sX}\abs{g(x)} \E P_{X\mid\sG}(x) = \sum_{x\in\sX}\abs{g(x)}P_X(x) = \E\abs{X} < \infty.\) ◻

Remark 9. The conditional characteristic function is given by \(\Phi_{X\mid\sG}(u) = \E[e^{juX}\mid\sG] = \int_{x\in\R}e^{jux}dF_{X\mid\sG}(x).\)

Definition 16. The conditional distribution of random variable \(X\) given random variable \(Y\) is defined as \(F_{X\mid Y}(x) \triangleq P(A_X(x)\mid\sigma(Y)]\) for all \(x\in \R\).

Example 17 (Conditional distribution given simple random variables). Consider a random variable \(X: \Omega \to \R\) and a simple random variable \(Y:\Omega\to\sY\) defined on the same probability space. Since random variables \(F_{X\mid Y}(x) = \E[\Ind{A_X(x)}\mid Y]\) are \(\sigma(Y)\) measurable, they can can be written as \(F_{X\mid Y}(x) = \sum_{y\in\sY}\beta_{x,y}\Ind{E_y}\) for some \(\beta_x\in\R^\sY\) and \(E_y = Y^{-1}\set{y}\) for all \(y \in \sY\). Further, we have \(\E[F_{X\mid Y}(x)\Ind{E_y}] = \E[\Ind{A_X(x)}\Ind{E_y}]\) for any \(y \in \sY\), which implies that \(\beta_{x,y} =\frac{P(A_X(x)\cap E_y)}{P_Y(y)} = F_{X\mid E_y}(x)\) for any \(y \in \sY\). It follows that \(F_{X\mid Y}\) is a \(\sigma(Y)\) measurable simple random variable.

Example 18 (Conditional expectation). Consider a random experiment of a fair die being thrown and a random variable \(X:\Omega\to\R\) taking the value of the outcome of the experiment. That is, for outcome space \(\Omega = [6]\) and event space \(\sF = \cP(\Omega)\), we have \(X(\omega) = \omega\) with \(P_X(x) = 1/6\) for \(x \in [6]\). Define another random variable \(Y = \SetIn{X \le 3}\). Then the conditional expectation of \(X\) given \(Y\) is a random variable given by Since \(P\set{Y = 1} = P\set{Y = 0} = 0.5\), it follows that that \(\E[\E[X|Y]] = \E[X] = 3.5\).

Example 19 (Conditional distribution). Consider the zero-mean Gaussian random variable \(N:\Omega\to\R\) with variance \(\sigma^2\), and another independent random variable \(Y \in \set{-1,1}\) with PMF \((1-p, p)\) for some \(p \in [0,1]\). Let \(X = Y + N\), then the conditional distribution of \(X\) given simple random variable \(Y\) is where \(F_{X\mid Y^{-1}(\mu)}\) is \(\int_{-\infty}^xe^{-\frac{(t-\mu)^2}{\sigma^2}}dt\).

Definition 20. When \(X, Y\) are both continuous random variables, there exists a joint density \(f_{X,Y}(x,y)\) for all \((x, y) \in \R^2\). For each \(y \in \sY\) such that \(f_Y(y) > 0\), we can define a function \(f_{X\mid Y}: \R^2 \to \R_+\) such that

Exercise 21. For continuous random variables \(X,Y\), show that the function \(f_{X\mid Y^{-1}(y)}\) is a density of continuous random variable \(X\) for each \(y\in \R\).